The UK Data Service

Who we are

  • Five partner universities

    • UK Data Archive, University of Essex (lead partner)
    • Cathie Marsh Institute, University of Manchester
    • Jisc, University of Manchester
    • EDINA, University of Edinburgh
    • University College London

  • 90+ staff

  • Since 2012 (UKDA \(\rightarrow\) 1967); curates national data since 2003

Custom footer

What we do

  • The main single point of access for UK social science data
  • Secondary data collection, curation and access
  • Training and user support
  • Communication and user engagement
  • Impact
  • … Key part of the UK social science research infrastructure, funded by the UKRI/ESRC

Our data…

  • UK social survey microdata:

    • Cross-sectional: large government and academic surveys
    • Longitudinal: major studies following people over time

  • International data: survey data, aggregate databases

  • Census tables and individual data – current and historical

  • Business microdata and administrative data

  • Qualitative data: multimedia files and interview transcripts

Our training in practice

  1. Webinars and online workshops
  2. User Conferences: four main user conferences each year
  3. Drop-in sessions: Survey, Computational Social Science and SecureLab
  4. Online learning materials: find key resources on our Learning Hub
  5. Helpdesk for individual data queries
  6. Check out our YouTube channel

1. Why a webinar series on data linkage?

Changing data landscape

  • Once upon a time: mostly census and surveys (lots!)
  • Emerging / new kinds of data in the last 20 years
    1. Administrative data
    2. Digital trace data (i.e. social media, web…)
    3. Smart data (i.e. flow, device generated)
    • 2 and 3: genuinely new data
    • 1: Digitalisation of records \(\longrightarrow\) greater availability
  • Increased demand for personal insights: the Monitored Self
  • Potential for new research avenues at a lower cost…

Growing role of non-survey data

  • Why interest is growing
    • New and previously unavailable measurements
    • Large scale, high frequency, low marginal cost
    • Particularly in social epidemiology and socio-economic research
    • Attractiveness of ‘harder’ type of data
  • But…
    • Collected for administrative purposes, not research
    • Selective coverage and population exclusions
    • Measurement error, changing definitions, policy artefacts
    • Limited socio-demographic and subjective information

Surveys are changing…

  • Rising costs under tighter budgets
  • Increasing fieldwork complexity
  • Recruitment challenges (reach, refusals, panel fatigue)
  • Alleviation comes at a cost: larger samples, incentives
  • Growth of online and mixed-mode designs

… But they remain essential:

  • Only source for attitudes, beliefs, motivations, well-being
  • Rich socio-demographic information not captured elsewhere: detailed occupation, social class, ethnicity
  • Theory-driven, validated instruments (ex: GHQ)
  • Only tested tool for population representative data, hard to reach groups and subgroup analysis

Increasing number of actors

  • Large academic data producers: CLS; ISER; ONS; NatCen
  • Linkage intermediaries: LLC, ADR, SAIL; SDR
  • Data curator: UKDS
  • Other consortiums: CLOSER
  • Government departments; health and education providers
  • \(\longrightarrow\) not always easy for researchers and analysts to navigate the landscape

Conclusion so far

  • Budgetary pressures on survey data
  • Wealth of cheaper, but narrowly focused, often unrepresentative new forms of data
  • Data integration enables validation and enhancement of both kinds of data (Benzeval 2020)
  • Increased complexity of the data provision landscape: especially for non experts
  • Linkage is still limited (but growing) practice and few linked datasets are available for secondary research
  • Need to adapt the skills training/capacity building

2. Exploring
integrated data

Working definition

  • Combining different sources of data ie:

    • Survey data \(\leftrightsquigarrow\) survey data
    • Survey data \(\leftrightsquigarrow\) non survey data
    • Non survey data \(\leftrightsquigarrow\) non survey data

  • That includes a shared unit of observation (individual, household, area…)

  • … In a coherent way in order to:

    • Validate or
    • .. enhance the original data

  • Bidirectional

  • In this presentation: linkage = integration

Validation example

  • How reliable are population survey-based estimates of chronic diseases?
  • Data linkage to validate prevalence of selected chronic conditions:
    • Angina, myocardial infarction, heart failure, and asthma
  • Link 11,323 adults from the 2013 and 2014 Welsh Health Survey to clinical data
  • Secure Anonymised Information Linkage (SAIL) Databank
  • Results: quality depends on condition:
    • Less agreement for cardiovascular, better for asthma
    • Potentially cheaper
    • But not devoid of techncial difficulties

Linking survey with digital trace data

  • Increasing usage of linked survey and social media data
  • Typical example: asking survey respondents to have their SM behaviour (digital trace) tracked
  • May reduce the cost of the survey (fewer questions to ask)
  • Subject to consent: representativeness issues
  • Consent depend on the app, gender, etc..
  • DIGISURVOR project for an exemple of current research
    • Linkage of existing survey data with online participation in political discussions

Kinds of data linked to survey data

Administrative data

  • Usually arising from the interaction between:

    • A public organisation or body…
    • … the unit for which records are produced (ie people)

  • Exemples:

    • Registry data: birth, death, marriage records,
    • Health records, educational transcripts
    • Government records: benefits, earnings/income
    • Financial reports ie credit ratings, mortgage application

  • In the UK: enabled Digital Economy Act 2017:

    “… de-identified data from government service providers, excluding NHS data, as part of their day-to-day functions, may be shared for public good research”

Hospital episodes data

  • NHS data about all hospital admissions in England.
  • Four datasets:
    • Episodes of using: Accident and Emergency, Admitted Patient Care, Adult Critical Care, Outpatients
    • Mostly available for 2007/9-2023
  • Data on diagnosis, maternity, mortality, mental health, treatment’s length, deprivation etc.
  • Available for the NCDS Birth Cohort

School inspection data

  • OFSTED ‘State of the nation’: anonymised data on latest schools inspections outcomes of 22,000 open schools

  • Linked with the MCS, currently covers years 2005 to 2019

  • Data on a wide range of topics i.e.:

    • Quality of teaching, learning and assessment
    • Effectiveness of leadership and management
    • Pupils’ achievement (aggregated) (2005-2015)
    • Behaviour and safety of pupils (2005-2015)

NEST pension data

  • National Employment Saving Trust ie ain occupational (i.e. employer led) pensions scheme for UK employees

  • Covers 1,000,000 employers, 11 millions employees

  • Data about:

    • Employer and employee characteristics
    • Current pension status
    • Pension contributions characteristics

Smart data

  • Fuzzier definition
    • Digital records held by private sector organisations
    • Measured data
    • Not traditionally associated with social research
    • Flow/ quasi real time data
  • Examples:
    • Data from fitness trackers, smartwatches, loyalty cards, purchase records and in-car smart tech: health and mobility behaviiour
    • Financial transactions by businesses and individuals < banks, supermarkets: financial behaviour and resilience
    • Energy network data (distribution & consumption); EV usage and charging; smart meter readings: building/households energy consumptions; net zero targets

Bio measurement data

  • Measured as part large scale longitudinal surveys (often via nurse visit):

    • Blood sample
    • Epigenetic data
    • Cortisol levels

  • Is this integrated data?

3. Which survey data are most commonly linked?

Linked survey data: in theory

  • Depends on:

    • The topic covered by the data linked i.e. does it match common topics studied in surveys?
    • The survey itself (i.e. does it include the required linking information / user consent)
    • … Scope of the surveys i.e. is linkage part of the original data collection, or is it a subsequent project?
    • Means available from the data producer

… And in practice

  • Major longitudinal studies:

    • Birth cohort studies
    • Next Steps and ELSA
    • Understanding Society

  • A few large scale cross-sectional surveys such as:

    • ASHE (Annual Survey of Hours and Earnings)
    • Family Resources Survey
    • Scottish Health Survey (project)

Birth cohort studies

  • Follow a sample of individuals from birth onwards
  • Four so far: people born in 1958(NCDS), 1970(BCS), 2000 (MCS), 2026 (Generation New Era)
  • Millenium Cohort Study (MCS)
    • ~ 19,000 children (born between June 2001 and Jan 03)
    • 7 ‘sweeps’: 9 months then at 3, 5, 7, 11, 14, years old
    • Parent and child interviews
    • Focuses on education, skills and health, truancy, cognitive ability, biological measurements
    • … Traditional socio-economic and demographic data

Other cohort studies

  • Next Steps

    • AKA Longitudinal Study of Young People in England
    • 16,000 people in England born 1980-90, from secondary school age (i.e. 13-14) onwards
    • Set up by DfE to study determinants of school outcomes

  • ELSA (English Longitudinal Study of Ageing):

    • Follows a sample of 19,000 people aged over 50 to understand all aspects of ageing in England.
    • Started in 2002, biennial waves.
    • Data on physical and mental health (incl. well-being), financial circumstances, and attitudes about ageing.

Understanding Society

  • Largest longitudinal study of the UK population

  • Initial sample size: 40K households, 100K individuals

  • 14 waves so far: 2009-23. Includes BHPS data 1991-2009

  • Ethnic minority boost samples, innovation panel

  • Very wide range of topics covered:

    • Family, partnerships, caring responsibilities,
    • Expenditure, consumption, deprivation
    • Social attitudes, values, political opinions
    • Transport, mobility, and commuting patterns
    • Environmental behaviours, and related attitudes

Annual Survey of Hours and Earnings (ASHE)

  • Produced by the Office for National Statistics
  • Large sample, drawn from NI records: 135,000-190,000 individuals per year
  • Small number of variables
  • Very precise source of information for pay components and working hours
  • Can be linked to other business surveys, as well as PAYE and pensions data
  • Some data available via Administrative Data Research UK

Integrated datasets curated by UKDS

Next Steps: Student Loans Data

  • Data on higher education loans for Next Steps participant

    • who provided consent to linkage in the age 25 sweep.

  • Information about:

    • Full Next Steps dataset +
    • applications for student finance,
    • payment transactions posted to participant’s accounts,
    • repayment details and
    • overseas assessment details.

    Hospitalisation Episodes Data (SN8681)

MCS: National Pupil Database

  • Data for children in England whose carer gave consent

  • Data from National Pupil Database and the Pupil Level Annual School Census.

    • Pupil level school census data from N1 to year 11 (2016/17)
    • KS1, KS2, KS4 and KS5 results (Years 2, 6, 11, 12 and 13)
    • Absence data from year 1 to year 11
    • School characteristics and school changes: N1 to year 11
    • Anonymised School identifiers (URN) and anonymised Local Education Authorities (LEA)

  • Also available for Next Steps and Understanding Society

    Ofsted Reports (SN9436)-

Vacancy Survey 2005-2025

  • Statutory, monthly survey of ~6,000 GB businesses
  • Single question:
    • “How many job vacancies for which actively seeking recruits from outside their organisation?”
  • Sample drawn from the Inter-Departmental Business Register (HMT, collected from VAT and PAYE registers)
  • Via linkage ISCO code (industrial activities classification), number of employees
  • Additional linkage via IDBR possible - including ASHE

4. Data integration and skills requirements

Survey analysis skills

  • Traditional deterministic matching (ie merging)
    • The simplest case: individual level data matched to individual level data non ambiguous identifier
    • The same holds to aggregate level (for example smart sensor small area level matching)
  • Probabilistic matching
    • When separate ids
    • When data is not clean
  • Statistical inference & non random samples

Computational skills

  • Non-survey cleaning (Pandas, Tidyverse)
  • Web scrapping (Python/R)
  • API queries (social media app: X, Reddit…)
  • Pattern detection ie random forest

Regulatory knowlege

  • Data protection & GDPR - rpe;requisite <- UKDS Safe researcher
  • Departmental regulations in case of Governmental data
  • Institutional/procedural: ie how to engage with the matching interface

4. Who’s who in the data integration landscape

Administrative Data Research (ADR)

  • Consortium of organisations, including the ONS, devolved governments and academic partners
  • Mission:
    • “link and open up de-identified data
    • generated from people’s interactions with public services,
    • making it securely available to accredited researchers.
  • Point of access for new data linkage within the public sector and between the public sector and researchers

UK Longitudinal Linking Consortium

  • Trusted Research Environment - TRE

  • Currently enables linkage between longitudinal studies and data from:

    • NHS England
    • Neighbourhood geographies
    • Address geographies

  • Planned: NHS Wales, Department for Work and Pensions, HM Revenue and Customs

UKDS

  • Collection, curation and access to linked survey data
  • Main gateway for secondary survey data analysis
  • Curates some linked data
  • Trusted Research environment

Data producers

  • Data producers of the main longitudinal studies ie
    • Understanding Society (ISER)
    • Main cohort studies (CLS)
  • Involved in data matching (ea consent management)
  • Closer - coordination & cross studies harmonisation
  • Government departments and the ONS

Secure Anonymised Information Linkage -SAIL

  • Wales based, but data from across the UK
  • Trusted Research Environment - TRE
  • Provides access to (mostly) health-related admin data
  • Enables data linkage between some of these to accredited researchers

6. Data linkage at UKDS: roles, routes, and researcher options

The UK Data Service and data linkage:
core principles

  • UKDS does not create linkages or integrate data
  • Linked data are created by data owners or processors
  • UKDS negotiates access to these data collections and makes them research-ready and safely accessible
  • The type of linkage researchers can undertake depends on:
    • the access level (Open / Safeguarded / Controlled)
    • the presence or absence of identifiers

What researchers can do in UKDS SecureLab

  • Researchers can:
    • Access more granular variables​
    • Create derived or contextual linkages, for example:​
    • Environmental or pollution deciles based on postcode-derived measures​
    • Area-level deprivation or service access indicators​
    • Import external datasets subject to depositor approval​
  • Key considerations:
    • UKDS SecureLab does not host direct identifiers
    • Researchers cannot create linkage spines or perform identifier-based matching​
    • All linkage activity must be explicitly approved as part of the project

7. How to access linked data at UKDS

References

Millennium Cohort Study: Linked Education Administrative Datasets (National Pupil Database - KS1-KS5), England, 2003-2021: Secure Access

Next Steps: Linked Administrative Datasets (Student Loans Company Records), 2007 - 2021: Secure Access

Vacancy Survey, 2005-2025: Secure Access

Grant, P. (2024) The Monitored Self In: The Virtual Hospital Springer, Cham.

Kerry- Barnard, S., Mohamad Zaki, N.H., Gomes, D., Ploubidis, G., Sanchez-Galvez, A. (2025) National Child Development Study: A guide to the linked health administrative datasets – Hospital Episode Statistics (HES). User Guide (Version 3). London: UCL Centre for Longitudinal Studies.

Peters, A., Sanchez-Galvez, A., Fitzsimons, E., Gomes, D. (2025) Millennium Cohort Study: Linked education administrative datasets-Ofsted User Guide (Version 1) London: UCL Centre for Longitudinal Studies.

Silber, H., Breuer, J., Beuthner, C., Gummer, T., Keusch, F., Siegers, P., … Weiß, B. (2022). Linking Surveys and Digital Trace Data: Insights From two Studies on Determinants of Data Sharing Behaviour Journal of the Royal Statistical Society, Series A (Statistics in Society), 185(Suppl. 2), 387-407.

Whiffen, T; Akbari, A ; Paget, T ; Lowe, S; Lyons, R (2020) How effective are population health surveys for estimating prevalence of chronic conditions compared to anonymised clinical data?, International Journal of Population Data Science (IJPDS) Vol 5:1